# Transformer Architecture

MobileLLM-125M is an autoregressive language model developed by Meta, using an optimized transformer architecture specifically designed for resource-constrained device applications. This model integrates several key technologies, such as the SwiGLU activation function, a deep thin architecture, shared embeddings, and grouped query attention. MobileLLM-125M/350M achieved accuracy improvements of 2.7% and 4.3%, respectively, on zero-shot commonsense reasoning tasks compared to the previous 125M/350M state-of-the-art models. The design principles can be effectively scaled to larger models, with MobileLLM-600M/1B/1.5B all achieving state-of-the-art results.

Aya Expanse is an open-weight research model with advanced multilingual capabilities. It merges high-performance pre-trained models with one year of research from Cohere For AI, including data arbitrage, multilingual preference training, safety tuning, and model fusion. This powerful multilingual large language model serves 23 languages, including Arabic, Chinese (Simplified and Traditional), Czech, Dutch, English, French, German, Greek, Hebrew, Hindi, Indonesian, Italian, Japanese, Korean, Persian, Polish, Portuguese, Romanian, Russian, Spanish, Turkish, Ukrainian, and Vietnamese.

Janus is an innovative autoregressive framework that achieves unified multimodal understanding and generation through the separation of visual encoding. This decoupling alleviates the role conflict of the visual encoder in understanding and generation tasks, enhancing the flexibility of the framework. Janus goes beyond previous unified models, matching or exceeding the performance of task-specific models. Its simplicity, high flexibility, and effectiveness make it a strong candidate for next-generation unified multimodal models.

Janus is an innovative autoregressive framework that addresses the limitations of previous methods by decoupling visual encoding into distinct pathways while utilizing a single, unified transformer architecture for processing. This decoupling not only alleviates the role conflict of the visual encoder in understanding and generation but also enhances the framework's flexibility. Janus outperforms earlier unified models and matches or exceeds the performance of task-specific models. Its simplicity, high flexibility, and effectiveness make it a strong candidate for next-generation unified multimodal models.

Skywork-Reward-Gemma-2-27B is an advanced reward model based on the Gemma-2-27B architecture, specifically designed for preference handling in complex scenarios. It has been trained on 80K high-quality preference pair data from multiple fields, including mathematics, programming, and safety. The model ranked first on the RewardBench leaderboard in September 2024, showcasing its strong capabilities in handling preferences.

Segment Anything Model 2 (SAM 2) is a visual segmentation model launched by Meta's AI research division, FAIR. It achieves real-time video processing through a simple transformer architecture and streaming memory design. The model builds a loop data engine through user interaction, gathering the largest video segmentation dataset to date, SA-V. SAM 2 is trained on this dataset, delivering outstanding performance across a wide range of tasks and visual domains.